Generation apparatus, system and method for generating virtual viewpoint image

ABSTRACT

To make it possible to obtain a natural virtual viewpoint image in which a structure or the like existing within an image capturing scene is represented three-dimensionally so as to be the same as a real one while suppressing a network load at the time of transmission of multi-viewpoint image data. The generation device according to the present invention generates a virtual viewpoint image based on three-dimensional shape data corresponding to an object, three-dimensional shape data corresponding to a structure, background data corresponding to a background different at least from the object and the structure, and information indicating a virtual viewpoint.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of International Patent ApplicationNo. PCT/JP2018/045977, filed Dec. 13, 2018, which claims the benefit ofJapanese Patent Application No. 2017-239888, filed Dec. 14, 2017, bothof which are hereby incorporated by reference herein in their entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a technique to generate an image from avirtual viewpoint based on a multi-viewpoint image captured from aplurality of viewpoint positions.

Background Art

In recent years, a virtual viewpoint image technique has been attractingattention, which reproduces an image from a camera that does not existactually (virtual camera) arranged virtually within a three-dimensionalspace by using images captured by a plurality of real cameras. Accordingto the virtual viewpoint image technique, for example, it is madepossible to view a highlight scene in a game, such as soccer orbasketball, from a variety of angles, and therefore, it is possible togive a user a stronger feeling of being at a live performance

For generation of a virtual viewpoint image, concentration of image datacaptured by a plurality of real cameras to an image processing serverand the like and generation and rendering processing of athree-dimensional model (shape data of an object) in the server and thelike may be necessary.

As a method of estimating a three-dimensional shape of an object, amethod called “visual volume intersection method (shape-from-silhouettemethod)” is known (PTL 1).

CITATION LIST Patent Literature

PTL 1 Japanese Patent Laid-Open No. 2014-10805

SUMMARY OF THE INVENTION Problem to be Solved by the Invention

With the conventional technique to estimate a three-dimensional shape,for example, there is a concern that a three-dimensional model is notgenerated for a structure that is a stationary object, such as a soccergoal, existing in the image capturing range. The reason is that theobject that is a target of estimation of a three-dimensional shape is aforeground portion, such as a person, which is a moving object withinthe image capturing range. That is, a structure in a still state, suchas a soccer goal, is handled as a background, and therefore, it is not atarget of generation of a three-dimensional model. In a case where avirtual viewpoint image is generated in a state where athree-dimensional model of a structure is not generated, the structureor the like that is not moving is represented two-dimensionally behind aperson or the like, so that the structure or the like is represented asif it were pasted onto the ground surface or the like, resulting in avideo image representation far from an actual image capturing scene. Anexample thereof is shown in FIG. 1. FIG. 1 is a virtual viewpoint imageof one scene of soccer and the image is such that the soccer goal(including all the elements of the goal post, the crossbar, and the goalnet) is pasted onto the turf field. Further, FIG. 13B is a virtualviewpoint image of one scene of sumo and the image is such that a sumowrestler who was pushed out of the sumo ring and fell down from the sumoring lies down on the sumo ring.

The present invention has been made in view of the above-describedproblems and an object thereof is to make it possible to obtain anatural virtual viewpoint image in which even a structure or the like,which is a stationary object, is represented three-dimensionally so asto become close to an actual one.

Means for Solving Problem

The system according to the present invention includes: a firstgeneration unit configured to generate, based on a plurality of capturedimages obtained by image capturing from a plurality of directions, afirst image representing a structure area and an object area beingdistinguished from other areas; a second generation unit configured togenerate, based on a plurality of captured images obtained by imagecapturing from a plurality of directions, a second image representingthe object area being distinguished from other areas except for theobject area; a transmission unit configured to transmit the first imagegenerated by the first generation unit and the second image; a firstacquisition unit configured to acquire, based on the first image and thesecond image both transmitted from the transmission unit,three-dimensional shape data corresponding to the object; a secondacquisition unit configured to acquire, based on the first imagetransmitted from the transmission unit, three-dimensional shape datacorresponding to the structure; a third acquisition unit configured toacquire background data corresponding to an area of a backgrounddifferent at least from the object and the structure; a fourthacquisition unit configured to acquire information indicating a virtualviewpoint; and a third generation unit configured to generate a virtualviewpoint image based on the three-dimensional shape data correspondingto the object acquired by the first acquisition unit, thethree-dimensional shape data corresponding to the structure acquired bythe second acquisition unit, the background data acquired by the thirdacquisition unit, and the information indicating the virtual viewpointacquired by the fourth acquisition unit, and the transmission unittransmits the first image at a frequency lower than that of the secondimage.

Advantageous Effect of the Invention

According to the present invention, it is possible to obtain a naturalvirtual viewpoint image in which even a structure or the like, which isa stationary object, is represented three-dimensionally so that thestructure or the like becomes close to an actual one.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram explaining a problem of a conventional method;

FIG. 2 is a diagram showing an example of arrangement of camera systemsaccording to a first embodiment;

FIG. 3 is a diagram showing an example of a hardware configuration of avirtual viewpoint image generation system;

FIG. 4 is a diagram explaining a common image capturing area of aplurality of cameras;

FIG. 5 is an explanatory diagram of volume data;

FIG. 6 is a sequence diagram showing a generation process of a structuremodel according to the first embodiment;

FIG. 7A is a diagram showing a captured image of a field in a statewhere there is no soccer goal;

FIG. 7B is a diagram showing a captured image of a field in a statewhere there is a soccer goal;

FIG. 8 is a diagram showing a three-dimensional model of a soccer goalon volume data;

FIG. 9 is a sequence diagram showing a generation process of a virtualviewpoint image according to the first embodiment;

FIG. 10A is a diagram showing an example of a captured image;

FIG. 10B is a diagram showing an example of a foreground image;

FIG. 10C is a diagram showing an example of a virtual viewpoint image;

FIG. 11 is a diagram showing a three-dimensional model of a player onvolume data;

FIG. 12 is a diagram showing an example of arrangement of camera systemsaccording to a modification example of the first embodiment;

FIG. 13A is a diagram explaining a problem of a conventional method;

FIG. 13B is a diagram explaining a problem of a conventional method;

FIG. 14 is a diagram showing that the periphery of a sumo ring isdivided into four areas on a bird's eye diagram in a case where the sumoring is viewed from directly above;

FIG. 15 is a flowchart showing a flow of processing to thin out andtransmit image data of a structure portion within an image capturingscene according to a second embodiment; and

FIG. 16 is a flowchart showing a flow of generation processing of avirtual viewpoint image according to the second embodiment.

DESCRIPTION OF THE EMBODIMENTS

In the following, with reference to the attached drawings, aspects forembodying the present invention are explained. Configurations shown ineach embodiment are merely exemplary and the present invention is notlimited to the configurations shown schematically.

First Embodiment

In recent years, due to improvement of image quality of a camera, theresolution of a captured image increases and there is a trend for thedata amount thereof to increase. In a case where multi-viewpoint imagedata captured by a plurality of cameras is transmitted as it is at thetime of transmitting the multi-viewpoint image data to a server or thelike via a network, a heavy load is imposed on the network. Further, thecalculation amount at the time of three-dimensional model generation andrendering processing in the server or the like having received themulti-viewpoint image data also increases. Consequently, in the presentembodiment, an aspect is explained in which a natural virtual viewpointimage is obtained, in which a structure or the like existing within theimage capturing scene is represented three-dimensionally so as to becomeclose to an actual one, while suppressing the network load at the timeof transmission of the multi-viewpoint image data. Specifically, anaspect is explained in which a structure whose still state or state nearto the still state continues within the image capturing scene isseparated as an object of a unique attribute, neither foreground norbackground, and a three-dimensional model of the structure is generatedin advance. In the following, explanation is given by taking a case asan example where the soccer game is taken as an image capturing sceneand a three-dimensional model of a soccer goal as a structure isgenerated in advance.

The virtual viewpoint image is a video image that is generated by an enduser and/or an appointed operator or the like freely operating theposition and orientation of a virtual camera and also called afree-viewpoint image, an arbitrary viewpoint image, and the like.Further, the virtual viewpoint image that is generated or themulti-viewpoint image that is the source of the virtual viewpoint imagemay be a moving image or a still image. In each embodiment described inthe following, an example of a case is explained mainly where both themulti-viewpoint image that is input and the virtual viewpoint image thatis output are moving images. The structure in the present embodiment isonly required to be a static object (stationary object) whose positiondoes not change in a case where image capturing is performed in a timeseries from the same angle. For example, in a case where an indoorstudio is taken to be an image capturing scene, it is possible to handlefurniture or a prop as a structure in the present embodiment.

FIG. 2 is a diagram showing arrangement of a total of ten camera systems110 a to 110 j configuring a virtual viewpoint image generation systemin a bird's eye diagram in a case where a field 200 is viewed fromdirectly above. Each of the camera systems 110 a to 110 j is arranged ata predetermined height from the ground so as to surround the field 200and acquires multi-viewpoint image data whose viewpoint is differentfrom one another by capturing the portion in front of one of goals froma variety of angles. On the turf field 200, a soccer court 201 is drawn(in fact, by white lines) and soccer goals are placed on both the leftside and the right side thereof. Further, a × mark 203 in front of asoccer goal 202 on the left side indicates a common line-of-sightdirection (gaze point) of the camera systems 110 a to 110 j and a brokenline circle 204 indicates an area that each of the camera systems 110 ato 110 j can capture with the gaze point 203 as a center. In the presentembodiment, it is assumed that a position is represented in a coordinatesystem in which one of the corners of the field 200 is taken as theorigin, the long side direction as an x-axis, the short side directionas a y-axis, and the height direction as a z-axis.

FIG. 3 is a diagram showing an example of the hardware configuration ofthe virtual viewpoint image generation system. The virtual viewpointimage generation system in FIG. 3 includes the camera systems 110 a to110 j, a switching hub 120, a control device 130, a server 140, and adatabase 150.

Within each of the camera systems 110 a to 110 j, image capturing units111 a to 111 j each including a lens, an imaging sensor, and the likeand camera adaptors 112 a to 112 j each performing control of the imagecapturing unit and predetermined image processing in accordance withinstructions of the control device 130 are included. The camera adaptorincludes a calculation processing device (CPU or ASIC) and memories (RAMand ROM) necessary for control and image processing. Further, the camerasystems 110 a to 110 j are connected by a daisy chain method in whichthe adjacent camera systems are connected by each of network cables 160a to 160 i. Image data captured by the camera systems 110 a to 110 j istransmitted via the network cables 160 a to 160 i. The switching hub(hereinafter, described as “HUB”) 120 performs routing of datatransmission on a network. The HUB 120 and the camera system 110 a areconnected by a network cable 170 a and the HUB 120 and the camera system110 j are connected by a network cable 170 b. The server 140 generatesvirtual viewpoint image data by modifying the multi-viewpoint image datatransmitted from the camera systems 110 a to 110 j. Further, the server140 is also in charge of the synchronization control of the entiresystem by generating a time synchronization signal. The database(hereinafter, described as “DB”) 150 accumulates the image data sentfrom the server 140 and provides the accumulated image data to theserver 150 as needed. The HUB 120 and the server 140 are connected by anetwork cable 170 c, the server 140 and the DB 150 are connected by anetwork cable 170 d, and the HUB 120 and the control device 130 areconnected by a network cable 170 e. The control device 130 centralizedlycontrols each of the camera systems 110 a to 110 j and the server 140.Then, the control device 130 outputs the virtual viewpoint imagegenerated by the server 140 based on the multi-viewpoint image to, forexample, a display device, not shown schematically, and anotherinformation processing apparatus on the network. In the systemconfiguration shown in FIG. 3, the plurality of camera systems isconnected by the daisy chain method, but the star connection in whichthe HUB 120 and each of the camera systems 110 a to 110 j are connecteddirectly with each other may be adopted. Further, the number of camerasystems configuring the virtual viewpoint image generation system is notlimited to ten.

Here, acquisition of the multi-viewpoint image data in the presentembodiment is explained. First, the server 140 transmits the timesynchronization signal to each camera system (time server function). Ineach of the camera systems 110 a to 110 j, each of the image capturingunits 111 a to 111 j performs image capturing in accordance with thereceived time synchronization signal under the control of the internalcamera adaptors 112 a to 112 j. Due to this, it is made possible toacquire the multi-viewpoint image by a moving image synchronized inunits of frames. Specifically, as described below, the image datacaptured by each camera system is sequentially transmitted to the server140. First, in the camera system 110 a, after image processing, to bedescribed later, is performed for the image data captured by the imagecapturing unit 11 a by the camera adaptor 112 a, the image data istransmitted to the camera system 110 b via the network cable 160 a. Thecamera system 110 b performs the same processing and transmits thecaptured image data to the camera system 110 c along with the capturedimage data acquired from the camera system 110 a. The same processing isperformed in each camera system and the captured image datacorresponding to a total of ten viewpoints, which is acquired by each ofthe ten camera systems 110 a to 110 j, is transmitted to the USB 120 viathe network cable 170 b and sent to the server 140 finally. The server140 performs generation of a structure model, to be described later,shape estimation of the object, and image processing, such as rendering,by using the received captured image data corresponding to tenviewpoints.

FIG. 4 is a diagram schematically showing an image capturing area ofeach of the image capturing units 111 a to 111 d possessed by each ofthe four camera systems 110 a to 110 d of the above-described ten camerasystems on the basis of FIG. 2 described previously. Each of triangularareas 411 to 414 extending from each of the camera systems 110 a to 110d is an image capturing area corresponding to each of the camera systems110 a to 110 d, represented by a visual volume. Then, a polygonal area415 in which the above four triangular image capturing areas 411 to 414overlap represents a common image capturing area of the camera systems110 a to 110 d. Here, the common image capturing area is explained bytaking a case of the four camera systems as an example, but it ispossible to derive the common image capturing area in a total of tencamera systems by the same method. As a matter of course, the commonimage capturing area in a total of ten camera systems is smaller thanthe polygonal area 415 described above. As describe above, it ispossible to obtain the common image capturing area of a camera groupcapturing the common gaze point by calculating the overlap area of thevisual volume possessed by each camera. Further, it is also possible tosimilarly derive the three-dimensional model of an object existing inthe common image capturing area from the overlap area of themulti-viewpoint image acquired by each camera system.

Next, a method of generating a three-dimensional model of a structureexisting within the common image capturing area obtained as describedabove is explained, which is one of features of the present embodiment.Here, explanation is given by taking a case where a three-dimensionalmodel of the soccer goal 202 is generated as an example. First, volumedata (see FIG. 5) in which the three-dimensional space on the field 200is filled with cubes (voxels) having a predetermined size is prepared.The value of the voxel configuring the volume data is represented by 0and 1 and “1” indicates an area contributing to shape formation and “0”indicates an area does not contributing to shape formation,respectively. In FIG. 5, symbol 501 indicates a voxel (drawn larger thanthe actual one for convenience of explanation). Next, thethree-dimensional coordinates of the voxel are converted from the worldcoordinate system into the camera coordinate system by using the cameraparameters of the image capturing units 111 a to 111 j included in eachof the camera systems 110 a to 110 j. Then, in a case where there is astructure in the camera coordinate system, a model (structure model)representing the three-dimensional shape of the structure by voxels isgenerated. The camera parameters refer to information on theinstallation position and orientation (line-of-sight direction) of eachof the image capturing units 111 a to 111 j, the focal distance of thelens, and the like.

FIG. 6 is a sequence diagram showing a generation process of a model ofa structure existing within an image capturing scene. The series ofprocessing shown by the sequence diagram is performed in advance beforethe start of image capturing (for example, before the start of a game)of a main part of a multi-viewpoint image, which is source data of avirtual viewpoint image, for example, at the time of the setup of thesports stadium, and the like. In FIG. 6, the set of the ten camerasystems 110 a to 110 j is described as “camera system group”.

At step 601, each of the image capturing units 111 a to 111 j capturesthe target three-dimensional space (here, the field 200) in the statewhere there is no structure (here, the soccer goal 202 is not installedyet). FIG. 7A shows an image obtained by the image capturing unit 111 iof the camera system 110 i capturing the field 200 in the state withoutthe soccer goal 202. The captured image such as this, whose viewpoint isdifferent from one another, is acquired in each camera system.

Next, at step 602, each of the image capturing units 111 a to 111 jcaptures the target three-dimensional space (field 200) in the statewhere there is a structure (here, the soccer goal 202 is installed).FIG. 7B shows an image obtained by the image capturing unit 111 i of thecamera system 110 i capturing the field 200 in the state with the soccergoal 202. As in the case with step 601, the captured image such as this,whose viewpoint is different from one another, is acquired in eachcamera system. It is assumed that the captured image data acquired atsteps 601 and 602 is stored in the memory within each of the cameraadaptors 112 a to 112 j.

At S603, each of the camera adaptors 112 a to 112 j separates the imagearea into the image area in which a structure is captured and the imagearea in which a background except for the structure is captured from thedifference between the captured image obtained at step 601 and thecaptured image obtained at step 602. Due to this, the image datacorresponding to the structure (here, the soccer goal 202) and the imagedata corresponding to the background (here, the field 200) except forthe structure are obtained.

At step 604, each of the camera adaptors 112 a to 112 j transmits theimage data corresponding to the structure and the image datacorresponding to the background, both obtained at step 603, to theserver 140.

At step 605, the server 140 generates a three-dimensional model of thestructure (here, the soccer goal 202) configured by voxels describedpreviously based on the image data of the structure received from eachcamera system and the camera parameters of each camera system. FIG. 8 isa diagram showing the three-dimensional model of the soccer goal 202 onthe volume data described previously. It may also be possible torepresent the three-dimensional shape by a set of points (point cloud),each indicating the center of a voxel, in place of the voxel itself. Thestructure model thus generated is stored in the memory or the DB 150within the server 140. Further, the background image data received alongwith the structure image data is also stored together.

The above is the flow of the processing at the time of generating astructure model within an image capturing scene. It may also be possibleto generate a three-dimensional model of another structure, for example,such as a corner flag, by the same method. In the present embodiment,separation of the structure and the background except for the structureis performed on the side of the camera adaptor, but is may also bepossible to perform separation on the side of the server 140.

Following the above, generation of a virtual viewpoint image in which astructure existing within an image capturing scene is representedwithout a sense of incongruity by using the structure model obtained asdescribed above is explained. FIG. 9 is a sequence diagram showing ageneration process of a virtual viewpoint image according to the presentembodiment. As in the case with the sequence diagram in FIG. 6, the setof the ten camera systems 110 a to 110 j is described as “camera systemgroup”.

In accordance with the start of a soccer game, or the like, at step 901,the control device 130 sends instructions to capture a multi-viewpointimage (image capturing start command), which is the source of a virtualviewpoint image, to the server 140. At step 902 that follows, uponreceipt of the image capturing instructions from the control device 130,the server 140 transmits the time synchronization signal to each of thecamera systems 110 a to 110 j. Then, at step 903, each of the camerasystems 110 a to 110 j starts image capturing of the targetthree-dimensional space (here, three-dimensional space on the field200). Due to this, for example, in the camera system 110 i, an imageduring a soccer game as shown in FIG. 10A is obtained. Then, imagecapturing of the image such as this, whose viewpoint is different fromone another, is performed in each camera system.

At step 904, in each of the camera adaptors 112 a to 112 j, processingto extract data of the foreground including moving objects (here, playerand ball) from the captured image acquired at step 903 is performed.This extraction processing can be said in other words as the processingto separate the captured image into the foreground and the backgroundbased a difference obtained by comparing the captured image acquired atstep 903 with the captured image (FIG. 7B) including the structure,which is acquired at step 602 described previously. FIG. 10B shows theimage of only the foreground, which is extracted from the captured image(whole image) in FIG. 10A. At step 905 that follows, each of the cameraadaptors 112 a to 112 j transmits the image data of the extractedforeground to the server 140. At this time, the image area (image dataof background) corresponding to the field 200 and the soccer goal 202 isnot transmitted to the server 140. By doing so, the data transmissionamount is suppressed accordingly.

At step 906, based on user instructions, the control device 130transmits instructions to generate a virtual viewpoint image (generationstart command) to the server 140 along with the information relating tothe virtual viewpoint and the gaze point. At this time, a user whodesires to create and view a virtual viewpoint image inputs informationnecessary for generation of a virtual viewpoint image via a GUI (notshown schematically) included in the control device 130. Specifically, auser sets information necessary for generation of a virtual viewpointimage (hereinafter, called “virtual viewpoint information”), such as theposition and the moving path of the virtual viewpoint, and further,which (which object) is gazed at, via a predetermined UI screen.

At step 907, the server 140 generates a three-dimensional model(foreground model) of a moving object within the image capturing sceneby using the image data of the foreground and the camera parametersdescribed previously, which are received from the camera group. Here,the three-dimensional models of the players and the ball are generatedas the foreground model. FIG. 11 is a diagram showing thethree-dimensional model corresponding to one certain player of thethree-dimensional models of the players and the ball generated at thisstep on the volume data as in FIG. 8 described previously.

At step 908, the server 140 generates a virtual viewpoint image by usingthe virtual viewpoint information received from the control device 130,the foreground model acquired at step 907, and the structure model andthe background data generated and acquired in advance. Specifically, theshape of each of the structure model and the foreground model in a casewhere they are viewed from a set virtual viewpoint (virtual camera) isestimated by using, for example, the shape-from-silhouette method. As aresult of the shape estimation processing, volume data representing thethree-dimensional shape of an object existing within the image capturingscene is obtained. In a case where the three-dimensional shape of anobject viewed from a virtual viewpoint is obtained, next, thethree-dimensional shapes of these objects are combined into one image.At the time of combination processing, in a case where the distancebetween the set virtual viewpoint and the foreground model is shorterthan that between the set virtual viewpoint and the structure model, theforeground model is mapped from above the structure model. On thecontrary, in a case where the structure model is nearer to the virtualviewpoint than the foreground model, the structure model is mapped fromabove the foreground model. In this manner, for example, the virtualviewpoint image in a case where the point to which the viewpoint fromthe image capturing unit 111 i of the camera system 110 i is moved inthe height direction (+z direction) is taken to be the virtual viewpointwill be the image as shown in FIG. 10C. In the virtual viewpoint imageshown in FIG. 10C, it is known that the players and the ball, which arethe foreground model, and the soccer goal, which is the structure model,are each mapped onto the field 200 in a natural three-dimensional shape.By repeating the processing such as this the number of timescorresponding to the number of time frames set separately, a desiredvirtual viewpoint image by a moving image is obtained.

In the present embodiment, an attempt is made to suppress the total datatransmission amount by not transmitting the background image data in theleast in the sequence in FIG. 9. In this case, for example, in capturinga moving image of a sports scene outdoors, the sunshine condition or thelike changes over time, and therefore, such a problem that thebackground portion in a finished virtual viewpoint image is differentfrom the actual one may arise. In a case where there is a concern ofsuch a problem, it may also be possible to appropriately transmitbackground image data obtained by the foreground/background separationat step 904 between transmissions of the foreground image data.

Further, in the present embodiment, generation of the structure modeland generation of the foreground model are performed by the server 140,but this is not limited. For example, it may also be possible for thecamera adaptor to generate the structure model and transmit thestructure model to the server 140. Alternatively, it may also bepossible for the server 140 to acquire the data of the structure modelgenerated by another information processing apparatus. What is requiredis that the state where it is possible for the server 140 to use thestructure model is brought about in the stage of generating theforeground model from the foreground data extracted from themulti-viewpoint image.

MODIFICATION EXAMPLE

In the above-described example, an attempt is made to reduce the datatransmission amount by handling the structure within the image capturingscene as an object of a unique attribute, neither foreground norbackground, and generating and storing in advance the three-dimensionalmodel of the structure. From the point of view of a reduction in thedata transmission amount, it is also possible to attain the object byhandling the three-dimensional model of the structure as a background.However, in a case where the structure model is handled as a background,the following problem occurs.

FIG. 12 is a diagram showing arrangement of a total of the ten camerasystems 110 a to 110 j configuring the virtual viewpoint imagegeneration system according to the present modification example in acase where the image capturing scene is sumo. Each of the camera systems110 a to 110 j is installed at the ceiling of the sumo venue so as tosurround the sumo ring and acquires multi-viewpoint image data whoseviewpoint is different from one another by capturing the sumo ring froma variety of angles. In this case, a three-dimensional model isgenerated based on the image obtained by capturing the sumo ring(=structure) alone and the obtained three-dimensional shape of the sumoring is handled as a background.

Here, it is assumed that, for example, one of sumo wrestlers fell downfrom the sumo ring as a result of the fight between the two sumowrestlers as shown in FIG. 13A. A case is considered where the state inFIG. 13A is captured by a total of the ten camera systems 110 a to 110 jand only the image data of the foreground is transmitted to the server140. The server 140 having received the image data of the foregroundmaps the two sumo wrestlers, who are the foreground, onto thethree-dimensional model of the sum ring created in advance as thebackground. As a result of this, an image will be obtained in which thewrestler who was pushed out and fell down from the sumo ring is lyingdown on the sumo ring. That is, in a case where the structure whosethree-dimensional model is generated is handled as a background, anatural virtual viewpoint image is not obtained depending on theposition of the foreground. Consequently, in a case where a structuremodel is handled as a background, it is desirable to determine inadvance whether a natural virtual viewpoint image is obtained and issuea warning to a user in a case where the possibility that an unnaturalvirtual viewpoint image is obtained is strong.

FIG. 14 is a bird's eye diagram in a case where the sumo ring is viewedfrom directly above and the periphery of the sumo ring is divided intofour areas A, B, C, and D. Each of the areas A, B, C, and D indicatesthe portion below the sumo ring (outside the sumo ring). A × mark at thecenter is the gaze point of the image capturing units 111 a to 111 jwithin the camera systems 110 a to 110 j. In the present modificationexample, in a case where instructions to generate a virtual viewpointimage are given, the position of the foreground is checked. In theabove-described example, whether the position of the sumo wrestler is onthe sumo ring is determined based also on the distance from thespecified virtual viewpoint (virtual camera) or the image of a camerathat captures a bird's eye view of the entire sumo ring, which is notshown schematically. Then, in a case where at least one of the wrestlersis not on the sumo ring and the position of the specified virtualviewpoint and the position of the wrestler do not exist in the same areaof any one of A to D, it is determined impossible to generate a virtualviewpoint image and a warning is issued. The reason is that in a casewhere the area in which the position of the virtual viewpoint exists andthe area in which the position of the sumo wrestler exists aredifferent, such as a case where one is within the A area and the otheris in the C area, the possibility that an unnatural virtual viewpointimage in which the sumo wrestler is pasted onto a position differentfrom the actual position is generated is strong. As described above, ina case where a structure model is handled as a background, it isnecessary to pay attention thereto.

According to the present embodiment, for the structure, thethree-dimensional model thereof is created in advance and thethree-dimensional model is handled differently from another foregroundmodel. Due to this, it is made possible to generate a virtual viewpointimage in which a structure within the image capturing scene isrepresented without a sense of incongruity while suppressing the datatransmission amount of the multi-viewpoint image that is the source ofthe virtual viewpoint image.

Second Embodiment

In the first embodiment, the aspect is such that the data transmissionamount is suppressed by separating the structure within the imagecapturing scene as an object of a unique attribute, which is neitherforeground nor background, and generating in advance thethree-dimensional model thereof and storing it in the server. Next, anaspect is explained as a second embodiment in which the datatransmission amount is suppressed by transmitting data of a structurewithin the image capturing scene after thinning the data thereof whilehandling the structure as a foreground. Explanation of the contents incommon to those of the first embodiment, such as the systemconfiguration, is omitted or simplified and in the following, differentpoints are explained mainly.

In the present embodiment also, explanation is given by taking the casewhere the soccer game is taken as the image capturing scene as anexample as in the first embodiment. That is, explanation is given belowon the premise that the arrangement of the camera systems is the same asin FIG. 2 described previously. In this case, the soccer goal, which isa structure, is handled as a foreground model, although beingdistinguished from the players and the ball. FIG. 15 is a flowchartshowing a flow of processing to transmit image data of a structureportion within the image capturing scene after thinning the image dataaccording to the present embodiment. The execution of the flow in FIG.15 is started in each camera system in a case where a user givesinstructions to capture a multi-viewpoint image, which is the source ofa virtual viewpoint image, via the UI of the control device 130. Thatis, the flow is implemented by the CPU or the like within the cameraadaptor executing a predetermined program.

Here, before the start of execution of the flow in FIG. 15, thepreparation processing thereof needs to be completed. Specifically, ineach of the camera systems 110 a to 110 j, the whole images (see FIG. 7Aand FIG. 7B) obtained by capturing the field 200 in the state wherethere is no structure and in the state where there is a structure areacquired in advance respectively and stored in the memory within each ofthe camera adaptors 112 a to 112 j. This preparation processing isperformed in advance at the time of the setup of the sports stadium, forexample, before a game is started. The data of these images obtained bythe preparation processing is transmitted also to the server 140 andstored in the memory within the server 140 for being referred to in thegeneration processing of a virtual viewpoint image, to be describedlater. On the premise of completion of the preparation processing suchas this, it is made possible to perform the flow in FIG. 15.

First, at step 1501, in each of the camera adaptors 112 a to 112 j, thevalue of a counter (not shown schematically) included therein isinitialized. Specifically, as the initial value, “0” is set. At step1502 that follows, in each of the image capturing units 111 a to 111 j,image capturing in accordance with the time synchronization signaltransmitted from the server 140 is started. Next, at step 1503,according to whether or not the current counter value is “0”, thefollowing processing is branched. In a case where the counter value is“0”, the processing advances to step 1507 and in a case where thecounter value is a value other than “0”, the processing advances to step1504.

At step 1504, “1” is subtracted from the counter value (the countervalue is decremented). At step 1505 that follows, in each of the cameraadaptors 112 a to 112 j, processing to extract the foreground area fromthe image (frame) captured by each of the image capturing units 111 a to111 j is performed. Specifically, processing to find a difference fromthe captured image (foreground/background separation processing) isperformed by using the whole image with a structure of the whole imagesof two patterns acquired and stored in advance in the preparationprocessing. Here, in the whole image with a structure of the wholeimages of two patterns acquired in the preparation processing, thesoccer goal 202 as a structure is captured in the state where the soccergoal 202 is installed on the field 200 (FIG. 7B). Consequently, theimage obtained by cutting out the area in which only the moving objects,such as players and the ball, are captured not including the soccer goalis obtained as foreground data. Then, at step 1506, each of the cameraadaptors 112 a to 112 j transmits the foreground data not including thestructure, which is obtained at step 1505, to the server 140. Aftertransmission of the foreground data is completed, the processingadvances to step 1510 and whether image capturing is terminated isdetermined. In a case where instructions to terminate image capturingare not received from the server 140, the processing returns to step1503.

At step 1507, in each of the camera adaptors 112 a to 112 j, processingto extract the foreground area from the image (frame) captured by eachof the image capturing units 111 a to 111 j is performed. Specifically,foreground/background separation processing to find a difference from acaptured image is performed by using the whole image with no structureof the whole images of two patterns, which are acquired and stored inadvance in the preparation processing. Here, in the whole image with nostructure of the whole images of two patterns, which are acquired in thepreparation processing, only the field 200 in the state where the soccergoal 202 is not installed yet is captured (FIG. 7A). Consequently, theimage is obtained as foreground data, in which not only the area inwhich the players and the ball are captured but also the area in whichthe soccer goal is captured is also cut out. That is, at this step, thesoccer goal, which is a structure, is also extracted as a foreground.Then, at step 1508, each of the camera adaptors 112 a to 112 j transmits“foreground data including a structure” obtained at step 1507 to theserver 140. At this time, transmission is performed by givinginformation indicating the presence/absence of a structure (for example,a binary flag indicating a case where a structure is included by “1” anda case where no structure is included by “0”) so that the side of theserver 140 having received the data learns that the area of thestructure is also included in the foreground data. At step 1509 thatfollows, a predetermined value N (N>1) is set to the counter.Specifically, in a case where the frame rate of moving image capturingby each of the image capturing units 111 a to 111 j is 60 fps, forexample, a value such as “60” is set. It is possible for a user tofreely change the frequency (once every N times) at which the foregrounddata including a structure is transmitted by setting the predeterminedvalue to be set to the counter to an arbitrary value. After setting ofthe predetermined value to the counter is completed, the processingadvances to step 1510 and whether image capturing is terminated isdetermined. In a case where instructions to terminate image capturingare not given by the server 140, the processing returns to step 1503.

The above is the contents of the processing to thin out and transmitimage data of the structure portion within the image capturing scene. Asa result of performing the processing such as this, for example, in acase where the same value as the value of the frame rate is set to thecounter as the predetermined value, the image data of the foregroundincluding a structure (here, soccer goal) is transmitted to the server140 only once every 60 times. Of course, the moving objects, such as theplayers and the ball, are transmitted at each of 60 times (for eachframe). As described above, it is possible to transmit image informationon a structure, which is a stationary object, at a frame rate reducedcompared to that of the moving objects, such as the players and theball, and therefore, it is possible to significantly increasetransmission efficiency compared to a case where the image data of theforeground including a structure is transmitted for each frame. Further,by transmitting the foreground image including image information on astructure at a frequency lower than that of the foreground image notincluding image information on a structure, it is possible to reducedata to be transmitted.

Next, processing at the time of generating a virtual viewpoint image bythe server 140 based on the image data of the foreground sequentiallysent as described above is explained. FIG. 16 is a flowchart showing aflow of generation processing of a virtual viewpoint image in the server140. The flow in FIG. 16 is performed in units of frames for theforeground image as a target, which corresponds to a specific time frame(for example, corresponding to ten seconds) specified by a user, fromthe image data of all the foregrounds captured by and transmitted fromeach of the camera systems 110 a to 110 j. The series of processing isimplemented by the CPU within the server 140 executing a predeterminedprogram based on instructions of the control device 130.

First, at step 1601, of the image data of the foreground correspondingto the set time frame, a foreground image (frame) of interest, which isa processing target, is determined. At step 1602 that follows, whether astructure is included in the foreground image of interest is determinedbased on the binary flag described previously. In a case wheredetermination results indicate that a structure is included in theforeground image of interest, the processing advances to step 1603 andin a case where no structure is included, the processing advances tostep 1605.

At step 1603 in a case where a structure is included in the foregroundimage of interest, an image area corresponding to the structure isextracted from the foreground image of interest and an imagerepresenting the structure (hereinafter, called a “structure image”) isgenerated. This generation processing is performed by a procedure asfollows. First, the difference between the captured image (whole image)in the state where there is a structure, which is acquired and stored inadvance in the preparation processing described previously, and theforeground image of interest is found, and the image area correspondingto the foreground is taken out. Next, the image area corresponding tothe foreground that is taken out and the captured image (whole image) inthe state where there is no structure stored in advance are combined.Then, the difference between the combined image obtained by thecombination and the foreground image of interest is found and thestructure image representing only the image area corresponding to thestructure is obtained. Then, at step 1604, the data of the structureimage generated at step 1603 is stored in the memory within the server140. In a case where the data of the structure image is already stored,the data is overwritten (updated) by the data of the structure imagegenerated anew. After the data of the generated structure image isstored in the memory, the processing advances to step 1607.

On the other hand, at step 1605 in a case where no structure is includedin the foreground image of interest, the data of the structure imagegenerated and stored in the processing at preceding step 1603 and step1604 is read. At step 1606 that follows, the read structure image andthe foreground image of interest including no structure are combined andthe foreground image of interest including the structure is generated.

At step 1607, a three-dimensional model (foreground model) of the objectwithin the image capturing scene, which takes the structure as a part ofthe foreground, is generated. At this time, at the step in a case wherea structure is included originally in the foreground image of interest(Yes at step 1602), the foreground model is generated by using theforeground image of interest as it is. On the other hand, at the step ina case where no structure is included originally in the foreground imageof interest, the foreground model is generated by using the foregroundimage of interest with which the structure is combined at step 1606 isgenerated. In any case, the foreground model including also the soccergoal, which is a structure (stationary object), in addition to themoving objects, such as the players and the ball, is generated.

At step 1608, based on the position information on the virtual viewpointset separately by a user, the virtual viewpoint image is generated byestimating the shape in a case where the foreground model generated atstep 1607 is viewed from the virtual viewpoint.

The above is the contents of the generation processing of a virtualviewpoint image in the server 140 according to the present embodiment.It is possible to obtain the same effect as that in the first embodimentalso by transmitting the image data of a structure within the imagecapturing scene after thinning the image data while handling thestructure as a foreground as in the present embodiment.

Other Embodiments

It is also possible to implement the present invention by processing tosupply a program that implements one or more functions of theabove-described embodiments to a system or an apparatus via a network ora storage medium and to cause one or more processors in a computer ofthe system or the apparatus to read and execute the program. Further, itis also possible to implement the present invention by a circuit (forexample, ASIC) that implements one or more functions.

The present invention is explained so far with reference to theembodiments, but it is needless to say that the present invention is notlimited to the embodiments described above. The scope of the followingclaims is to be accorded the broadest interpretation so as to encompassall such modifications and equivalent structures and functions.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

What is claimed is:
 1. A generation apparatus comprising: a firstacquisition unit configured to acquire three-dimensional shape datacorresponding to an object captured from a plurality of directions; asecond acquisition unit configured to acquire three-dimensional shapedata corresponding to a structure captured from a plurality ofdirections; a third acquisition unit configured to acquire backgrounddata corresponding to a background different at least from the objectand the structure both captured from a plurality of directions; a fourthacquisition unit configured to acquire information indicating a virtualviewpoint; and a generation unit configured to generate a virtualviewpoint image based on the three-dimensional shape data correspondingto the object acquired by the first acquisition unit, thethree-dimensional shape data corresponding to the structure acquired bythe second acquisition unit, the background data acquired by the thirdacquisition unit, and the information indicating the virtual viewpointacquired by the fourth acquisition unit, wherein the second acquisitionunit acquires the three-dimensional shape data corresponding to thestructure by generation thereof based on a captured image obtained byfirst image capturing in a state where the object and the structure donot exist and a captured image obtained by second image capturing fromthe same direction as that of the first image capturing in a state wherethe structure exists and the object does not exist.
 2. The generationapparatus according to claim 1, wherein the second acquisition unitacquires the three-dimensional shape data corresponding to the structureby generation thereof before starting image capturing of the object. 3.The generation apparatus according to claim 1, wherein thethree-dimensional shape data corresponding to the structure is generatedbased on an image which is based on a captured image obtained by thefirst image capturing and a captured image obtained by the second imagecapturing and which displays the structure area being distinguished fromother areas.
 4. The generation apparatus according to claim 1, whereinthe generation unit: combines, in a case where a distance from a virtualviewpoint specified by the information indicating the virtual viewpointacquired by the fourth acquisition unit to the object is shorter than adistance from the specified virtual viewpoint to the structure, thethree-dimensional shape data corresponding to the structure and thethree-dimensional shape data corresponding to the object so that theobject exists in front of the structure in the virtual viewpoint image;and combines, in a case where the distance from the specified virtualviewpoint to the structure is shorter than the distance from thespecified virtual viewpoint to the object, the three-dimensional shapedata corresponding to the object and the three-dimensional shape datacorresponding to the structure so that the object exists behind thestructure in the virtual viewpoint image.
 5. The generation apparatusaccording to claim 1, wherein the object is a moving object.
 6. Thegeneration apparatus according to claim 1, wherein at least one of aperson and a ball is the object.
 7. The generation apparatus accordingto claim 1, wherein the structure is an object whose still statecontinues.
 8. The generation apparatus according to claim 1, wherein atleast one of a soccer goal and a corner flag, which are used in a soccergame, is the structure.
 9. The generation apparatus according to claim1, wherein the structure is an object installed at a predeterminedposition.
 10. The generation apparatus according to claim 1, wherein atleast a part of the structure is installed on a field on which a person,who is an object, plays a game.
 11. The generation apparatus accordingto claim 1, wherein the structure is a specified object.
 12. A systemcomprising: a first generation unit configured to generate, based on aplurality of captured images obtained by image capturing from aplurality of directions, a first image representing a structure area andan object area being distinguished from other areas; a second generationunit configured to generate, based on a plurality of captured imagesobtained by image capturing from a plurality of directions, a secondimage representing the object area being distinguished from other areas;a transmission unit configured to transmit the first image generated bythe first generation unit and the second image generated by the secondgeneration unit; a first acquisition unit configured to acquire, basedon the first image and the second image both transmitted from thetransmission unit, three-dimensional shape data corresponding to theobject; a second acquisition unit configured to acquire, based on thefirst image transmitted from the transmission unit, three-dimensionalshape data corresponding to the structure; a third acquisition unitconfigured to acquire background data corresponding to an area of abackground different at least from the object and the structure; afourth acquisition unit configured to acquire information indicating avirtual viewpoint; and a third generation unit configured to generate avirtual viewpoint image based on the three-dimensional shape datacorresponding to the object acquired by the first acquisition unit, thethree-dimensional shape data corresponding to the structure acquired bythe second acquisition unit, the background data acquired by the thirdacquisition unit, and the information indicating the virtual viewpointacquired by the fourth acquisition unit, wherein the transmission unittransmits the first image at a frequency lower than that of the secondimage.
 13. The system according to claim 12, further comprising: afourth generation unit configured to generate, based on the first imageand the second image both transmitted from the transmission unit, athird image representing the structure area being distinguished fromother areas.
 14. The system according to claim 13, wherein the firstacquisition unit and the second acquisition unit are physically the sameand acquire, in a case where the second image is transmitted from thetransmission unit, the three-dimensional shape data corresponding to theobject and the three-dimensional shape data corresponding to thestructure based on the second image transmitted from the transmissionunit and the third image generated by the fourth generation unit. 15.The system according to claim 12, wherein the third generation unit:combines, in a case where a distance from a virtual viewpoint specifiedby the information indicating the virtual viewpoint acquired by thefourth acquisition unit to the object is shorter than a distance fromthe specified virtual viewpoint to the structure, the three-dimensionalshape data corresponding to the structure and the three-dimensionalshape data corresponding to the object so that the object exists infront of the structure in the virtual viewpoint image; and combines, ina case where the distance from the specified virtual viewpoint to thestructure is shorter than the distance from the specified virtualviewpoint to the object, the three-dimensional shape data correspondingto the object and the three-dimensional shape data corresponding to thestructure so that the object exists behind the structure in the virtualviewpoint image.
 16. A generation method of generating a virtualviewpoint image, the generation method comprising: a first acquisitionstep of acquiring three-dimensional shape data corresponding to anobject captured from a plurality of directions; a second acquisitionstep of acquiring three-dimensional shape data corresponding to astructure captured from a plurality of directions; a third acquisitionstep of acquiring background data corresponding to a backgrounddifferent at least from the object and the structure both captured froma plurality of directions; a fourth acquisition step of acquiringinformation indicating a virtual viewpoint; and a generation step ofgenerating a virtual viewpoint image based on the three-dimensionalshape data corresponding to the object acquired by the first acquisitionunit, the three-dimensional shape data corresponding to the structureacquired by the second acquisition unit, the background data acquired bythe third acquisition unit, and the information indicating the virtualviewpoint acquired by the fourth acquisition unit, wherein in the secondstep, the three-dimensional shape data corresponding to the structureare acquired by generation thereof based on a captured image obtained byfirst image capturing in a state where the object and the structure donot exist and a captured image obtained by second image capturing fromthe same direction as that of the first image capturing in a state wherethe structure exists and the object does not exist.
 17. A generationmethod of generating a virtual viewpoint image, the generation methodcomprising: a first generation step of generating, based on a pluralityof captured images obtained by image capturing from a plurality ofdirections, a first image representing a structure area and an objectarea being distinguished from other areas; a second generation step ofgenerating, based on a plurality of captured images obtained by imagecapturing from a plurality of directions, a second image representingthe object area being distinguished from other areas; a transmissionstep of transmitting the first image generated at the first generationstep and the second image generated at the second generation step; afirst acquisition step of acquiring, based on the first image and thesecond image both transmitted at the transmission step,three-dimensional shape data corresponding to the object; a secondacquisition step of acquiring, based on the first image transmitted atthe transmission step, three-dimensional shape data corresponding to thestructure; a third acquisition step of acquiring background datacorresponding to an area of a background different at least from theobject and the structure; a fourth acquisition step of acquiringinformation indicating a virtual viewpoint; and a third generation stepof generating a virtual viewpoint image based on the three-dimensionalshape data corresponding to the object acquired at the first acquisitionstep, the three-dimensional shape data corresponding to the structureacquired at the second acquisition step, the background data acquired atthe third acquisition step, and the information indicating the virtualviewpoint acquired at the fourth acquisition step, wherein in thetransmission step, the first image is transmitted at a frequency lowerthan that of the second image.